Skip to content

perf(db): skip unchanged index updates#1691

Merged
KyleAMathews merged 6 commits into
TanStack:mainfrom
K-Mistele:perf/skip-unchanged-index-updates
Jul 24, 2026
Merged

perf(db): skip unchanged index updates#1691
KyleAMathews merged 6 commits into
TanStack:mainfrom
K-Mistele:perf/skip-unchanged-index-updates

Conversation

@K-Mistele

@K-Mistele K-Mistele commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Index updates now skip bucket, tree, and timestamp writes when the indexed value is unchanged. The combined change also caches compiled index evaluators, avoids object-normalization work for primitive values, repairs a BasicIndex bookkeeping edge case, and adds model-based property coverage for both index types.

Root cause

Every update previously called the public remove/add paths for every index, even when the indexed value stayed in the same normalized bucket. That repeated expression compilation, normalization, map/set or B+ tree writes, and timestamp updates for changes to unrelated row fields.

The first no-op guard also exposed an existing partial-state case in BasicIndex: if removal expression evaluation failed, the value bucket could retain the key while indexedKeys dropped it. Checking only bucket membership could then return early without repairing keyCount.

Approach

  • Evaluate and normalize the old and new indexed values once.
  • Compare normalized values with SameValueZero semantics, matching JavaScript Map keys for values such as NaN and signed zero.
  • Return early only when the value is unchanged and the expected bucket contains the key. BasicIndex also confirms membership in indexedKeys.
  • Move changed values through private bucket helpers, avoiding duplicate evaluation and updating the timestamp once.
  • Preserve the existing public remove/add fallback when expression evaluation fails.
  • Compile each index expression once and reuse its evaluator.
  • Return primitive values from normalizeValue() before object-specific checks.
  • Check randomized valid mutation sequences against a reference model after every operation.

Key invariants

  • A no-op update leaves buckets, ordered entries, key counts, and timestamps unchanged.
  • A real move removes the key from the old bucket and adds it to the new bucket exactly once.
  • SameValueZero governs bucket identity; custom ordering comparators do not redefine equality.
  • undefined, null, NaN, signed zero, dates, byte arrays, object identity, and custom comparators retain their prior lookup semantics.
  • keyCount, indexed keys, exact buckets, ordered entries, equality lookups, range queries, and rebuilt state agree with the reference model.
  • Expression-evaluation failures retain the prior fallback behavior.

Non-goals

Trade-offs

The no-op path performs cheap membership checks before returning. This is slightly more work than comparing values alone, but prevents an optimization from hiding inconsistent bookkeeping.

A reverse key-to-bucket map would also solve stale oldItem values, but adds one map entry per indexed row per index and broadens this performance change into a separate correctness invariant. That work remains isolated in #1517.

Benchmark

The original #1691 microbenchmark used 10,000 unique indexed rows, four indexes, 5,000 updates, and five runs; medians are shown:

Index Workload main This PR
BasicIndex indexed value unchanged 293.6 ms 1.3 ms
BTreeIndex indexed value unchanged 274.7 ms 1.1 ms
BasicIndex indexed value changes 244.1 ms 141.4 ms
BTreeIndex indexed value changes 232.4 ms 129.1 ms

This is a focused index microbenchmark, not an end-to-end application result. The gain depends on index count, bucket shape, and how often indexed values change. Evaluator caching and the primitive normalization fast path were added after this measurement.

Verification

pnpm exec vitest run packages/db/tests/index-update.property.test.ts packages/db/tests/index-update-short-circuit.test.ts
pnpm --filter @tanstack/db test
pnpm --filter @tanstack/db build
pnpm exec eslint packages/db/src/indexes/base-index.ts packages/db/src/indexes/basic-index.ts packages/db/src/indexes/btree-index.ts packages/db/src/utils/comparison.ts packages/db/tests/index-update-short-circuit.test.ts packages/db/tests/index-update.property.test.ts
pnpm exec prettier --check .changeset/quick-trees-rest.md packages/db/src/indexes/base-index.ts packages/db/src/indexes/basic-index.ts packages/db/src/indexes/btree-index.ts packages/db/src/utils/comparison.ts packages/db/tests/index-update-short-circuit.test.ts packages/db/tests/index-update.property.test.ts
  • Focused update and property suite: 23 tests passed.
  • Full @tanstack/db suite: 2,483 tests passed, 5 skipped.
  • Package build, ESLint, and Prettier checks passed.

Files changed

  • base-index.ts: cache the compiled expression evaluator.
  • basic-index.ts: short-circuit unchanged updates, use direct bucket moves, and guard key bookkeeping.
  • btree-index.ts: short-circuit unchanged updates and use direct bucket moves.
  • comparison.ts: add SameValueZero equality and a primitive normalization fast path.
  • index-update-short-circuit.test.ts: cover no-op updates, real moves, edge values, comparators, and evaluation failures.
  • index-update.property.test.ts: compare randomized mutation sequences with a reference model.
  • quick-trees-rest.md: provide one combined patch changeset.

Related: #1517, #1645

Summary by CodeRabbit

  • Bug Fixes
    • Avoid unnecessary bucket writes on no-op updates; preserves timestamps and reuses bucket instances.
    • Improve indexed value comparison with SameValueZero semantics (e.g., NaN), while keeping undefined and null distinct.
    • Incrementally move keys between index buckets on changes, with consistent state even when index value evaluation fails.
  • Tests
    • Added cross-index update short-circuit tests and expanded failure/edge-case coverage.
    • Added property-based randomized put/delete invariants tests for update correctness.
  • Chores
    • Cache compiled index expression evaluation to speed up repeated updates.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Index updates now compare normalized indexed values, skip unchanged writes, and update bucket membership incrementally when values change. Both index implementations share bucket helpers and SameValueZero comparison semantics, while evaluator compilation is cached and tests cover edge cases, failures, and randomized invariants.

Changes

Index update optimization

Layer / File(s) Summary
Evaluator caching and value comparison
packages/db/src/indexes/base-index.ts, packages/db/src/utils/comparison.ts
Caches compiled index evaluators, preserves primitive normalization, and adds SameValueZero equality semantics.
Shared equality and bucket management
packages/db/src/indexes/basic-index.ts, packages/db/src/indexes/btree-index.ts
Delegates bucket insertion and removal to helpers while maintaining value mappings and ordered values.
Incremental update short-circuit
packages/db/src/indexes/basic-index.ts, packages/db/src/indexes/btree-index.ts
Skips unchanged normalized-value updates, moves keys when values change, and retains fallback handling for evaluation failures.
Update validation and release metadata
packages/db/tests/index-update-short-circuit.test.ts, packages/db/tests/index-update.property.test.ts, .changeset/quick-trees-rest.md
Tests edge cases, failures, bookkeeping, randomized index invariants, and records the patch release changeset.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant IndexUpdate
  participant Comparison
  participant IndexBuckets
  participant EvaluatorCache
  IndexUpdate->>EvaluatorCache: evaluate indexed expression
  EvaluatorCache-->>IndexUpdate: normalized old and new values
  IndexUpdate->>Comparison: compare normalized values
  Comparison-->>IndexUpdate: equality result
  IndexUpdate->>IndexBuckets: preserve bucket or move key membership
  IndexBuckets-->>IndexUpdate: updated index state
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and matches the main change: skipping unchanged index updates for db indexes.
Description check ✅ Passed The description is detailed and covers the change, motivation, approach, verification, and release context.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/db/src/utils/comparison.ts (1)

232-233: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Avoid propagating any through the new index-value boundaries.

Use unknown for arbitrary normalized values; equality checks and Map access do not require any.

  • packages/db/src/utils/comparison.ts#L232-L233: change equality parameters to unknown.
  • packages/db/src/indexes/basic-index.ts#L102-L103: change normalizedValue to unknown.
  • packages/db/src/indexes/basic-index.ts#L146-L147: change normalizedValue to unknown.
  • packages/db/src/indexes/basic-index.ts#L163-L164: change normalized update locals to unknown.
  • packages/db/src/indexes/btree-index.ts#L104-L105: change normalizedValue to unknown.
  • packages/db/src/indexes/btree-index.ts#L141-L142: change normalizedValue to unknown.
  • packages/db/src/indexes/btree-index.ts#L160-L161: change normalized update locals to unknown.

As per coding guidelines, “Avoid using any types; use unknown instead when the type is truly unknown.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/utils/comparison.ts` around lines 232 - 233, Replace any with
unknown at all normalized index-value boundaries: update areSameValueZeroEqual
parameters in packages/db/src/utils/comparison.ts, normalizedValue parameters
and normalized update locals in packages/db/src/indexes/basic-index.ts (lines
102-103, 146-147, 163-164), and the corresponding parameters and locals in
packages/db/src/indexes/btree-index.ts (lines 104-105, 141-142, 160-161).
Preserve the existing equality checks and Map access behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/db/tests/index-update-short-circuit.test.ts`:
- Around line 40-54: Add an `undefined` to `null` case to the `it.each` table in
the “keeps the existing bucket” test area, asserting the update moves the key to
the `null` bucket rather than reusing or conflating the `undefined` sentinel
bucket; preserve the existing bucket-reuse assertions for genuinely equal
values.

---

Nitpick comments:
In `@packages/db/src/utils/comparison.ts`:
- Around line 232-233: Replace any with unknown at all normalized index-value
boundaries: update areSameValueZeroEqual parameters in
packages/db/src/utils/comparison.ts, normalizedValue parameters and normalized
update locals in packages/db/src/indexes/basic-index.ts (lines 102-103, 146-147,
163-164), and the corresponding parameters and locals in
packages/db/src/indexes/btree-index.ts (lines 104-105, 141-142, 160-161).
Preserve the existing equality checks and Map access behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ef2caabc-db57-4e0c-85f4-e1f1a1170788

📥 Commits

Reviewing files that changed from the base of the PR and between 7246c75 and 2cc26fd.

📒 Files selected for processing (5)
  • .changeset/quick-trees-rest.md
  • packages/db/src/indexes/basic-index.ts
  • packages/db/src/indexes/btree-index.ts
  • packages/db/src/utils/comparison.ts
  • packages/db/tests/index-update-short-circuit.test.ts

Comment thread packages/db/tests/index-update-short-circuit.test.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/db/src/utils/comparison.ts`:
- Around line 232-233: Remove the local areSameValueZeroEqual implementations
from the BasicIndex and BTreeIndex classes, and import the shared
areSameValueZeroEqual utility from packages/db/src/utils/comparison.ts instead.
Update all references in both index implementations to use the imported function
while preserving existing SameValueZero equality behavior.
- Around line 232-233: Update areSameValueZeroEqual to replace the rejected
self-comparison NaN check with Number.isNaN(a) && Number.isNaN(b), while
preserving strict equality behavior and SameValueZero semantics.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1008310c-dccd-4ddd-ac8f-5e44838f9321

📥 Commits

Reviewing files that changed from the base of the PR and between 2cc26fd and b693fbb.

📒 Files selected for processing (4)
  • packages/db/src/indexes/basic-index.ts
  • packages/db/src/indexes/btree-index.ts
  • packages/db/src/utils/comparison.ts
  • packages/db/tests/index-update-short-circuit.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • packages/db/tests/index-update-short-circuit.test.ts
  • packages/db/src/indexes/btree-index.ts
  • packages/db/src/indexes/basic-index.ts

Comment thread packages/db/src/utils/comparison.ts Outdated
@pkg-pr-new

pkg-pr-new Bot commented Jul 24, 2026

Copy link
Copy Markdown
More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@1691

@tanstack/browser-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1691

@tanstack/capacitor-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1691

@tanstack/cloudflare-durable-objects-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1691

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@1691

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@1691

@tanstack/db-sqlite-persistence-core

npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1691

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@1691

@tanstack/electron-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1691

@tanstack/expo-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1691

@tanstack/node-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1691

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@1691

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1691

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@1691

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@1691

@tanstack/react-native-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1691

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1691

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@1691

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@1691

@tanstack/tauri-db-sqlite-persistence

npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1691

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1691

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@1691

commit: 8a9c859

@KyleAMathews
KyleAMathews merged commit 8ee783d into TanStack:main Jul 24, 2026
11 checks passed
@github-actions github-actions Bot mentioned this pull request Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants